European Journal of Foreign Language Teaching 
ISSN: 2537 - 1754 

ISSN-L: 2537 - 1754 

Available on-line at: www.oapub.org/edu 


doi: 10.5281/zenodo.437574 Volume 2 | Issue 1 | 2017 


A CRITICAL REVIEW OF RESEARCH ON CURRICULUM 
DEVELOPMENT AND EVALUATION IN ELT 


Esin Diindar, Ali Merc 
Anadolu University, Turkey 


Abstract: 

This study was designed to compile the studies conducted on curriculum development 
and evaluation in ESL/EFL contexts and to specify their general characteristics through 
content analysis. The studies were chosen in line with the inclusion criteria through 
which online articles and dissertations were included. No specific timeline, context or 
research design was set for the literature search. As a result of the review of the related 
literature, 86 studies were reached and analyzed in terms of their contexts, sample type 
and size, data collection tools, data analysis techniques. All the studies were coded 
through a protocol and the results were tabulated. As well as the characteristics of the 
studies, their findings were also analyzed through content analysis and common points 
were presented and interpreted. As a result of the analysis, it was revealed that the 
curriculum evaluation studies were more popular than the curriculum development 
studies. It was also found that the studies were mainly about primary and secondary 
levels; they mostly employed Likert-type scales; semi-structured interviews were more 
common; descriptive statistics were applied more frequently. Finally, common findings 
were detected among the studies conducted in different contexts with different grades 
of study. By taking the findings as the basis, some suggestions are provided for further 
studies. 
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Introduction 


In order to see the tendencies of the studies in a specific field, to organize the findings, 
and to see the points that have not been researched yet, researchers can make critical 
reviews on the general characteristics and findings of the studies on the same topic. 
This helps the researcher to examine the big picture of the field and see the tendencies 
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of the studies in a specific field. Understanding the content of the studies of a specific 
topic in the field can guide researchers and accelerate the further studies. 

Curriculum development and evaluation is one of the core topics in English 
language teaching. This issue is important and getting the attention of the researchers as 
it is directly related to the presentation of the language in a systematic way and 
teaching learning situations. Although the metaphor of manual can be used to describe 
the curriculum, designing a curriculum is not easy as just writing down what to teach 
and how to teach. Curriculum design and evaluation, as a part of the development 
process, is at the center of English teaching and other matters as it contains the 
approaches, methods, techniques, activities followed to teach the language as well as 
the content and it is based on a policy (Brown, 1995). 


1. Significance of the study 


First of all, considering the fact that no critical analysis study has been encountered in 
English language teaching literature on curriculum design and evaluation, this study is 
supposed to be an important source of information. Second, the present study has a 
substantial value as it combines and evaluates both quantitative and qualitative 
findings in one analysis in order to support the statistical data driven from the findings 
of the quantitative studies with qualitative data to draw a clear picture of the issue. 
Finally, the study reveals the tendency of the research on English language curriculum 
development and evaluation in ESL/EFL context by not only documenting the general 
characteristics of the studies, presenting statistical information about the most 
frequently used data collection tools, sample types and research designs, but also by 


summarizing the common findings of the related studies. 


1.1. Curriculum Development 

The number of the children and adults laboring over second or foreign language 
learning, which has been one of the biggest educational enterprise worldwide, has 
reached over millions (Richards, 2001, p.1). Not only the learners but also the teachers 
as components of the educational system put a lot of effort into this educational attempt 
through organizing the lessons, selecting or adapting the teaching materials and 
applying their plans in the classroom (Richards, 2001). No matter what approach the 
teachers follow, there is something universal in deciding on what to teach the learners 
and in every single approach it is kind of a rule to put the subjects to be taught in an 
order (Brown, 1995). At this point, it is crucial to mention an umbrella term, which 
contains approach, method, technique, syllabus, activity, and exercise within itself: 


curriculum. Curriculum means much more than subjects to be taught, transmitted or 
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delivered (Kelly, 2004, p. 1). An effective definition of the curriculum should provide 
the information on why are we teaching, what could be the possible effects of the 
transmission of the information, what are the outcomes (Kelly, 2004). Curriculum can 
be seen as a detailed manual for teaching and learning process: 

Curriculum refers to the specific blueprint for learning that is derived from 
desired results—that is, content and performance standards (be they state-determined 
or locally developed).Curriculum takes content (from external standards and local 
goals) and shapes it into a plan for how to conduct effective teaching and learning. It is 
thus more than a list of topics and lists of key facts and skills (the “input”). It is a map of 
how to achieve the “outputs” of desired student performance, in which appropriate 
learning activities and assessments are suggested to make it more likely that students 
achieve the desired results (Wiggins & McTighe, 2006, pp. 5-6). 

Drawing a distinction between education and curriculum, Null (2011) states that 
curriculum has a crucial place if how effective an institution is the question. The 
curriculum contains many questions within itself and Null (2011) lists the questions that 
curriculum holds within itself: “What should be taught, to whom, under what circumstances, 
how, and with what end in mind? Put more concretely, what should be taught to these students, 
in this school, at this time, how, and to what end? What process should we use to decide what 
our curriculum ought to be within a particular school, college, or university context?” (p. 5). 
Eisner (1994) classifies the curriculum as: (a) explicit curriculum which has the publicly 
explicit goals or opportunities provided by the schools and they are stated in 
curriculum guides of the schools or n the materials provided, (b) implicit or hidden 
curriculum, which stands for the intentional and unintentional functions of the school 
and these are not advertised contrary to the explicit curriculum, they just arise in the 
context through learning opportunities, and (c) null curriculum, which deals with 
intellectual processes and content neglected by the schools. Null curriculum is related 
to what is not taught in the school or not provided. From a different point of view, Kelly 
(2004, pp. 2-7) classifies curriculum as “educational, total, hidden, the planned and the 
received, the formal and the informal”. 

The main focus of curriculum development is on deciding which knowledge, 
skills and values to be taught, how to reach the intended outcomes, and the learning 
and teaching processes (Richards, 2001). An effective language curriculum is not just 
related to the pure action of teaching; it also includes the procedures of planning, 
designing and implementation (Richards, 1990). Constructed on the main principles of 
development, conducting and evaluation, curriculum development has six main steps: 
“needs analysis, goal setting, syllabus design, methodology, testing and evaluation” 
(Richards, 1990, p. 1). 
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Philosophical, theoretical and practical constructions give shape to the 
curriculum development; in other words, “science, society, moral doctrine, knowledge, and 
the learner” are the sources of the curriculum (Ornstein & Hunkins, 2009). Similar to 


Richards (1990), Brown (1995, p. 20) suggests that “needs analysis, objectives, testing, 


materials, teaching and evaluation” are the basic components of curriculum design. 


NEEDS ANALYSIS 
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Figure 1: Systematic Approach to Designing and Maintaining Language Curriculum 
(Brown, 1995, p. 20) 


As presented in Figure 1, the first step is needs analysis, the concept which focuses on 
the learners and concerns with the language structures which are likely to be needed 
(Brown, 1995). In language curriculum development needs analysis helps; (1) providing 
a systematic approach for the selection of the input, constructing the content of the 
program by taking the opinions of members of designing process, (2) specifying the 
language needs, (3) providing a base for the assessment of the present program 
(Richards, 1990). When it comes to the goals and objectives, we should make a clear 
distinction between these two terms. Goal is a more general term defining what should 
we do to meet the expectations of the learners and objective is a more specific term 
related to the structures that learners should know to reach a specific goal (Brown, 
1995). The objectives can be behavior, content, proficiency or skill based (Richards, 
1990). Goals and objectives, the second step rings the need for the third step language 
testing. In a language program, tests can be applied in the need for placement of the 


students, identifying the levels of the students through diagnostic tests, or testing the 
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achievements of the learners (Brown, 1995). Then it comes to the last step before the 
classroom implementation of the language curriculum, material design. 
Another model of curriculum design is suggested by Macalister and Nation 


(2011). The model, which is shown in Figure 2, is constructed with three outer and one 


inner circle, which is also divided into three sub-circles. 


Content 
and 


Monitoring 


Assessing 


Figure 2: A Model of the Parts of the Curriculum Design Process 
(Macalister & Nation, 2011, p. 2) 


Starting from the inner circle, the model puts the goals into the center of the curriculum 
design in order to emphasize their crucial role in a course and here the sequence and 
content represents what and in which order to teach, the part, format and presentation, 
is generally deals with how to present the language structures to the learners, the part 
we plan the lesson which includes the techniques and activities and the last component 
of the inner circle is monitoring and assessing in which we check the outcomes and 
evaluate the learning activity and the success of the teaching (Macalister & Nation, 
2011). 

Before giving information about the outer circles and what they stand for, it is 
important to mention that they all have sub-factors. Environment analysis can reveal 
the factors related to the “learners, teachers and teaching-learning situations”, needs 
analysis has tree sub-factors “lacks, wants and necessities” and the last one, principles, is 
divided into “content and sequencing, format and presentation, and monitoring and 
assessment” (Macalister & Nation, 2011, pp. 3-4). 

The last component of the model is evaluation, which encircles the whole model, 
and it can provide detailed information about every piece and component of the model 
and can show the lacks and necessities or the parts to be developed, and generally this 


component is neglected in curriculum development (Macalister & Nation, 2011). 
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1.2. Curriculum Evaluation 

After designing the curriculum, conducting the needs analysis and the actual 
implementation of the designed curriculum, the development is not totally over as there 
are still lots of things to be considered. We need to deal with some questions like 
whether the curriculum really meets the expectations of the people affected by it, or we 
can reach our objectives through this curriculum, what is the situation in the class in 
which we carry out the curriculum we designed, compared to the other examples, 
whether we achieve to design a better curriculum (Richards, 2001). In order to find an 
answer to all these questions or the concerns we have, we need to evaluate the whole 
process of our curriculum design. 

Compared to needs analysis, evaluation is a broader notion dealing with every 
process in the curriculum development from the specification of the objectives, to the 
design or adaptation of the materials and in-class implementation, and processing all 
the information gathered during the stages of development and application (Brown, 
1995). The concern of evaluation is not just the results. Weir and Roberts (1994) 
approach evaluation with some basic questions such as why, what, how long. There are 
sO many aspects we can evaluate in a curriculum such as the needs of the curriculum as 
a whole or of the learners, sources, the system including curriculum, in-class 
implementation, the achievement and motivation of the learners, the success of the 
school staff including teachers and principal, and the conditions under which learning- 
teaching situation is carried out (Weir & Roberts, 1994). 

The main reason behind conducting an evaluation is to provide beneficial 
information to a larger audience and a theoretical base and a context based information 
on particular implementations (Weir & Roberts, 1994). Literature provides some 
approaches to evaluate the curriculum. For example, Brown (1995, p. 219) summarize 
them under four main categories: ‘product oriented approaches’, which focus on the 
extent of reaching the objectives, static characteristic approaches, conducted by an 
outsider to evaluate the efficacy of the program, and ‘process oriented approaches’, 
which answer the questions of how to improve or revise the current curriculum, and 
decision facilitation approaches, based on collecting information before making 
decisions. When it comes to the types of evaluation there are two main types: formative 
and summative (Weir & Roberts, 1994; Brown, 1995; Richards, 2001; Nation & 
Macalister, 2010). Formative evaluation is conducted throughout the process and the 
main purpose of using formative evaluation is to gather data to improve the curriculum 
during the development and implementation of it (Weir & Roberts, 1994; Brown, 1995; 
Richards, 2001; Nation & Macalister, 2010). The data collected through a formative 
evaluation is mostly detailed, related to the process itself, and used for the 


improvement of the staff or the material and specifying the goals (Nation & Macalister, 
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2010). Formative evaluation is in a way checking the quality in each stage of curriculum 
development and getting the feedback regularly and thus, it provides justification for 
the changes made by the staff during the process (Weir & Roberts, 1994). Summative 
evaluation is conducted at the end of the process and it helps to determine to what 
extent the curriculum or the course is successful, it deal with how effective the 
curriculum is (Weir & Roberts, 1994; Brown, 1995; Richards, 2001; Nation & Macalister, 
2010). Brown (1995) criticizes summative evaluation as it ignores the fact of language 
program’s being a continuing structure and suggests that the administrators of the 
program can stop the process and ask the questions related to summative evaluation 
while the program is being applied. However, summative evaluation still has a crucial 
place as it helps to see the bigger picture, and gives information on what has been 
accomplished in a period of time. Together with formative evaluation, summative 
evaluation can save the school staff from the troubles and stress of being evaluated by 
an outsider (Brown, 1995). 

Another crucial aspect is how to conduct the evaluation. Post-modernity resulted 
in a paradigm shift and this has caused the redefinition of achievement and this shift 
puts more responsibility on the evaluation process in terms of giving effective feedback 
on the best choice under the policy based conditions (Kiely & Rea-Dickins, 2005). With 
this shift, the emphasis on basing the curriculum on an ideology has changed into an 
understanding of bearing the values, other dimensions and factors putting a great 
responsibility on the stakeholders, participants of the curriculum (Kiely & Rea-Dickins, 
2005). The determination of the criteria for the evaluation is another challenge. 
According to Kiely and Rea-Dickins (2005, p. 13), there are three approaches to decide 
on criteria: “theory based, policy based and constructivist or ethnographic approach” and as 
well as providing benefits, these approaches also make it hard to specify such a criteria 
that is covering both the experience of the participants in the program and its rationale. 
The other two challenges are dealing with a big amount of data, knowing what to do 
and where to use it, and as the results of the evaluation are expected to support or 
construct a theory or provide information for the other programs in a broader sense, it 
is also challenging for the stakeholders how to deal with results and report them (Kiely 
& Rea-Dickins, 2005). 

According to Nation and Macalister (2010, pp. 123-4) the steps of the evaluation 
are as follows: 

1) specify the audience of the evaluation and what they expect from this, 

2) specify the field in which the findings will be used, 

3) decide whether there is really a need for the evaluation, 

4) find out the time span and sources necessary for conducting the evaluation, 


5) specify the aspects to be evaluated in the program, 


European Journal of Foreign Language Teaching - Volume 2 | Issue 1 | 2017 142 


Esin Diindar, Ali Merg 
A CRITICAL REVIEW OF RESEARCH ON CURRICULUM DEVELOPMENT AND EVALUATION IN ELT 


6) create connections to get the help of the people in the system, 


7) specify the participants and data gathering tools, 


8) decide on how to report the evaluation results, 


9) check whether a follow up evaluation is appointed. 


There are numbers of tools for gathering information, presented in Table 1, 


which a scholar can choose in line with the focus of aim such as interviews, checklists 


and tests, observations, meetings, and self-reports (Weir & Roberts, 1994; Brown, 1995; 


Nation & Macalister, 2010). 


Table 1: Focus and Tools for Evaluation of Teaching and Learning 
(Nation & Macalister, 2010, p. 129) 


Focus 


Tools 


Amount of learning 


Quality of learning 


Quality of teaching 


Quality of course book 


Quality of curriculum design 


Degree of later success of graduates of the course 


Teacher, learner or sponsor satisfaction 


Achievement and proficiency tests 
Learner self-report scales 
Analysis of course book content 
Interviewing learners 


Achievement and proficiency assessment 
Lesson observation 

Interviewing learners 

Teacher diaries 

Study of research reports 


Systematic lesson observation 

Interviewing teachers-retrospective accounts 
Learner self-report scales 

Teacher self-report scales 

Study of research reports 

Achievement tasks 

Listing of staff qualifications 


Systematic coursebook evaluation checklist 
Teacher and learner questionnaire 


Systematic course evaluation checklist 
Analysis of the syllabus 
Evaluation of the course materials 


Interviewing employers or questionnaires 
Interviewing graduates or questionnaires 
Later achievement records such as GPA 


Self-report scales 
Questionnaires 

Interviews 

Learner re-enrollment statistics 
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1.3. Previous Studies on Curriculum Development and Evaluation 

Studies conducted on the issue of curriculum development and evaluation in 
different contexts has provided variety into the field. Some examples are in-class 
curriculum application in Taiwan elementary schools (Lai, 2007), the effect of learner 
driven motives on the development and implementation of the curriculum (Shawer, 
Gilmore, & Banks-Joseph, 2009), designation of a curriculum with art based medium for 
kindergarten level in Puerto Rico (Perez, 2009), the design of kindergarten English 
curriculum based on DAP assumptions (Sowers, 1996), the analysis of backward design 
process in foreign language curriculum (Korotchenko, Matveenko, Strelnikova, & 
Phillips, 2015), the investigation of English curriculum in Asia Pacific Region (Nunan, 
2003) as well as some curriculum evaluation studies (Abu-Ghararah, 1986; Alwan, 2006; 
Burgos, 2012; Harris, 2010; Hillberry, 2008; Hu, 2007; Krekeller, 1993; Powell, 2008; Sun, 
2007; Wang, 1996). 

In Turkey, the tendency is to evaluate the curriculum of English language 
courses from different levels along with the perspectives of teachers and students, and 
quite rarely the parents and inspectors. These are mainly about the evaluation of 2™4 
grade English curriculum (Kandemir, 2016; Kiigiiktepe, Ktictiktepe, & Baykin, 2014; 
Mavis & Bedir, 2014; Aybek, 2015; Oziidosru & Adigiizel, 2015; Yildiran & Tanriseven, 
2015), the evaluation of 3" grade curriculum (Cankaya, 2015), the evaluation of 4" and 
5" grade English curriculum (Er, 2006; Erkan, 2009; Giines, 2009; Mersinligil, 2002; 
Seckin, 2010), 6", 7, 8 grades (Celen, 2011; Demirlier, 2010; Orakc1, 2012; Ozer, 2012; 
Yanik, 2008; Yigit, 2010; Yorti, 2012), the difficulties experienced by the teachers during 
the application of English curriculum (Ari, 2014), the evaluation of 9" grade curriculum 
(Karci, 2012), CEFR-related curriculum (Zorba & Arikan, 2016), and development of 
English curriculum in Gulhane Military Medical Academy (Sari, 2003). Although there 
are several studies on the issue of curriculum development and evaluation in English 
language teaching, no critical analysis has been encountered in the literature. 

As the reviewed literature suggests, there is a need to have a better 
understanding of the current studies and their findings to be able to conduct more 
effective studies in future. In revealing the neglected parts of the issue, this study has a 
significant role. Instead of evaluating a curriculum through a Likert-type scale form the 
perspectives of the teachers and students, which is the general tendency of the studies 
especially in Turkey, drawing a general picture of the related literature will help us to 


make sense of the findings and meet the needs of the field. 


1.4. Research Questions 
The main objective of the present study is to conduct a critical review of the 


studies on curriculum development and evaluation conducted in ESL/EFL contexts, to 
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find out the general characteristics of the studies and to present a synthesis of the 

qualitative studies based on the same issues. In line with this aim, the study is an 
attempt to answer these research questions; 

1. What are the general characteristics of the studies on curriculum development 
and evaluation in ESL/EFL context? 

2. What is the general pattern of the findings of the studies on curriculum 


development and evaluation in ESL/EFL context? 
2. Method 


2.1. Research Design 

Many independent studies providing lots of different findings on any specific 
issue can sometimes be misleading; therefore, there is a need for a further, 
comprehensive and reliable research in order to interpret fund of knowledge (Demirel, 
2005; Ozcan, 2008). Card (2012) claimed that the need for organizing the existing studies 
is more urgent than conducting further studies in social sciences. This need has resulted 
in the combining the findings on the same issue and analyzing them within a one single 
study. In order to analyze the studies on curriculum development and evaluation in 
ELT, this study adapted document analysis, which involves the analysis of written 
documents presenting information on the phenomena to be investigated (Yildirim & 
Simsek, 2008). 


2.2. Data Collection Procedure 

The study was conducted through the following steps; (1) specification of the 
topic, (2) defining selection criteria, (3) searching for the studies, (4) specifying the final 
set of data, (5) coding the studies, (6) calculating descriptive statistics (7) tabulating and 


reporting the findings (8) interpreting the findings and making conclusions. 


2.2.1. Inclusion and exclusion criteria. 

When the researcher specifies the topic of the study, the second step before 
searching for the primary studies to be analyzed, is setting the inclusion/exclusion 
criteria. For this present study, these criteria were set; 

Timeline 

Not a specific period of time has been set for the present study. 

Publication type 

Articles and dissertations that are available online have been included in the study. 
Research design 


The studies with both quantitative and qualitative research design are included. 
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Context 
This study utilized the studies conducted in Turkey and in any country where English 


is taught as a second or foreign language. 


2.3. Data Collection 
The process of searching the literature conducted following the steps offered by 
Card (2012). The steps are given in Figure 3. 


1. Articulate sample frame 3. Plan search strategy 
Modify search strategy if inadequate 


2. Specify inclusion/exclusion 4. Computerized databases/ 5. Unpublished works 
criteria, 2.2. reference volumes e Conference programs 
e Constructs of interest e Database selection e Funding agency lists 
e Sample characteristics e Select key words/ e Research registries 
e Study design combination e E-mails/listservs 


6. Initial list of studies 7. Input from experts in field 
e Constructed while 
reviewing search results 
Modify criteria if unclear 


papiilreeersercame 11. Further input from 


experts in field 
; Proceed if adequate 
Proceed if avequate 


8. Forward searches : 5 
¢ Continue until low yield 12.Final list of studies 


9. Backward searches 10. Revised list of studies 
e Conduct while coding 


Figure 3: Basic Steps for Searching the Literature (Card, 2012, p. 35) 


As the first step, an overall literature search was conducted in order to construct the 
frame of the study. The second step was to specify the inclusion/exclusion criteria. At 
this point, the timeline, publication type, research design, statistical value and context 
were determined as inclusion/exclusion criteria. After the third step, which was 
organization of the search, the literature search was conducted through online 
databases. In this step, in order to specify the studies to include for the analysis, certain 
keywords were determined. According to Card (2012), the selection of the key words 
can be specified through the knowledge of the researcher either by investigating the key 
words of the studies on the issue to be analyzed or taking thesauri as a base. The key 


words used for the search were “curriculum development (f=11), curriculum evaluation 
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(f=10), English language curriculum (f-13), curriculum (f=10)”. As the studies in Turkey 
might have been written in the native language, the key words in Turkish were defined 
as “program gelistirme (f=6), program degerlendirme (f=7), Ingilizce dersi 6&retim programt 
(f=6)”. All these key words were determined based on the frequency among the key 
words of the studies found out during the overall search of the literature and they were 
checked through UNESCO IBE Glossary of curriculum terminology (2013). Next, online 
research databases were scanned to obtain the studies. In order to find the articles, 
Educational Resources Information Center (ERIC) was used as the search engine as it is 
one of the rich databases for educational journals. For the dissertations, YOK theses and 
dissertations database and Proquest were searched. After writing down the initial list of 
the studies, backward and forward searches were followed as the sixth and seventh 
steps. During these steps, the reference lists of the studies and the ones citing the 
studies reached were analyzed to find out more studies as well as to be certain that not 
all but most of the possible studies were reached. At the end, the initial list was revised 
and a final list of studies was prepared. As a result of the whole process, 86 studies 
were included for the present study (See Appendix): 24 articles (28%) and 62 theses and 
dissertations (72%). 


2.4. Coding the Studies 

After searching for the studies and specifying the ones to be analyzed in line 
with the inclusion/exclusion criteria, a coding protocol was designed based on the 
article classification form developed by Tatar, Kagizmanli, and Akkaya (2013). This 
coding protocol consists of two parts: the identity and the content of the study. The 
identity of the study describes the author, publication date, and the title of the studies 
while the content mainly deals with the sample types and size, research design, data 


collection tools, and data analysis. 


2.5. Data Analysis 

In order to reveal the general characteristics of the studies, the quantitative data 
were coded using the protocol form adapted from Tatar et al. (2013). The qualitative 
data, on the other hand, were analyzed through content analysis, which enables to 
summarize the data in a categorical and systematic way (Bttytkoztiirk, Cakmak, 
Akgiin, Karadeniz, & Demirel, 2011). The steps of the content analysis were as follows: 

1. The findings of the studies were listed down. 

All the findings were reviewed. 
The common answers were determined and grouped. 


The semantically identical ones were coded and categorized. 


SA aR 


All categories were given a theme. 
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3. Results and Discussion 


This critical review investigated the studies on curriculum development and evaluation 
in ESL/EFL for their context, grade, publication date, research design, sample type and 
size, data collection tools, and data analysis techniques. The first finding is the 
distribution of the studies based on the context that they had been conducted. As a 
result of the analysis, 19 different contexts were identified and their frequencies are 
presented in Table 2. As two different contexts were compared in some studies, the total 
number was found to be more than the number of the studies analyzed. It was revealed 
that nearly more than half of the studies analyzed were conducted in Turkey (51%) 
followed by Taiwan and USA (8%) and China (7 %). It would not be correct to interpret 
the high frequency of the studies of Turkey as a sign of leading the literature of 
curriculum development and evaluation. The present study mainly focused on the 
context of Turkey and Turkish key words were used during the literature search; 
therefore, the frequency of the studies conducted in Turkey outnumbered the ones in 


other contexts. 


Table 2: The Distribution of the Studies Based on Their Contexts 


Contexts f % 
Turkey 46 51 
Taiwan 7 8 
USA 7 8 
China 6 7 
Korea 4 4 
Saudi Arabia 4 4 
Japan 3 3 
Puerto Rico 3 3 
Thailand 2 y) 
Vietnam 1 1 
Hong Kong 1 1 
Nigeria 1 1 
Kuwait 1 1 
Malaysia 1 1 
Canada 1 1 
United Arab Emirates 1 1 
Libya 1 1 
Djibouti 1 1 
Brazil 1 1 
TOTAL 89 100 


As mentioned before, not a specific timeline was determined for the present study. The 
results of the analysis showed that the studies analyzed were published between 1985 
and 2016. Comparing the number of the publications according to their publication 
period, 67 studies were published the years between 2006 and 2016 (78%). There were 
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14 studies conducted during the period of 1995-2005 (16%), and there were only five 
(6%) studies in the years between 1985 and 1994. 


Table 3: Distribution of the Studies Based on Their Publication Date 


Publication Date f % 
1985-1994 5 6 

1995-2005 14 16 
2006-2016 67 78 
TOTAL 86 100 


The increase of the studies starting from the beginning of the 21% century can be 
explained with the curriculum innovations of the countries, especially in Turkey 
(Dénmez, 2010), China (Lee, 2007), and Taiwan (Chen, 2013). With the policy changes in 
Turkey, in 2006-2007 academic year English, was decided to be given starting from the 
4" srade. In 2012, it was taken to the 2" grade (Yildiran & Tanrisever, 2015). From the 
second half of the 19" century to 21* century, China has also made changes in its 
English language policy and started to give more importance to English language 
teaching day by day (Liu, 2015). Hence, there can be a correlation between the time of 
policy changes of the countries and the number of the studies conducted in the field of 
education. 

In Table 4, the sample types used in the studies are presented. According to 
results, the researchers mainly investigated the opinions of the teachers/instructors 
about the curriculum they were applying both in Turkey and in other ESL/EFL contexts. 
Table 4 also shows us that other stakeholders affected by the curriculum have been 
neglected in studies, especially parents. There is only one study in Turkey and two 
studies in other contexts including parents to the curriculum development and 
evaluation. As a part of the system, parents should be more involved in giving feedback 
and sharing opinions about curriculum development and evaluation processes. None of 
the analyzed studies in Turkey dealt with the officials, policy makers, or the program 
directors, the studies including these stakeholders in other contexts are quite rare. It is 
also evident that there are fewer studies conducted with students compared to the ones 
with teachers both in Turkey (31%) and other contexts (22%). 


Table 4: Distribution of the Sample Types 


Turkey Others 
Sample Types ; % 5 % 
Teachers/Instructors 38 62 31 47 
Administrators 2) 3 5 fi 
Parents 1 2 2 3 
Inspectors 1 2 0 0 
Supervisors 0 0 3 - 
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Program facilitators 0 0 1 D 
Program directors 0 0 2 3 
Officials 0 0 + 6 
Policy makers 0 0 1 2 
Coordinators 0 0 1 2 
Publishers 0 0 1 2 
Students 19 31 15 De; 
TOTAL 61 100 66 100 


Listening to the voice of the teachers is important as they are the ones who are applying 
the curriculum in a real classroom environment. They are good feedback sources for the 
policy makers. Here, the factor affecting the sample choice of the researchers can be that 
reaching a sample group especially the parents, inspectors or officials is not as easy as 
reaching the teachers or students. Thus, many researchers may have a tendency to use 
convenient sampling to conduct their studies. 

As it is presented in Table 5, more than half of the studies conducted in Turkey 
mainly dealt with 24, 34, 4%, and 5 grades (51%). Because more than one grade was 
evaluated in some studies, total number of the grades is different from the number of 
the studies. 28% of the studies investigated the curriculum of 6", 7, and 8" grades. In 
other words, the studies mostly focused on the primary and secondary school contexts 
in Turkey. High school (15%) and college (6%) are the least covered levels in Turkish 
context. For the other contexts, the studies covering the grades from kindergarten to 5 
grade and high school level have an equal percentage (26%) while 38% of the studies 
were dealing with 6", 7 and 8" grades. There were 11 studies about the curriculum of 


college level in other contexts (10%). 


Table 5: The Distribution of the Grades Investigated 


Grades Turkey Others 

f % f % 
Kindergarten 0 0 3 3 
1 0 0 4 + 
2 10 14 5 5 
3 1 1 6 5 
4 12 18 4 4 
5 12 18 6 5 
6 7 10 10 9 
7 6 9 15 14 
8 6 9 16 15 
9 4 6 1 11 
10 2 3 5 5 
11 2 3 5) 5 
12 2 3 6 5 
College 4 6 11 10 
TOTAL 68 100 108 100 
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The important point to be highlighted is that there is no study related to the 
kindergarten curriculum in Turkey. This can be explained within the fact that English is 
not an official course for kindergarten in Turkey yet although some private schools 
have been giving English courses at this level. We can state that curriculum design and 
evaluation for kindergarten level is a neglected issue both in Turkey and other contexts 
as a result of the educational policies of the countries. 

The distribution of sample sizes is presented in Table 6. The analysis showed that 
among 86 studies, 75 of them had a sample group and 11 of the studies conducted 
document analysis. According to Table 5, among the studies with a sample group, 19% 
of them have a sample size from 0 to 10, 17% of them have between 11-30, 11% of them 
have between 31-60, 9% of them have between 61-100, 31% of them have between101- 
500, and 13% of them have a sample size more than 500. Although no certain number is 
given for the sample size of the studies in the literature, it is “ideal” to have a sample 
size of 300-400 as it is “advantageous” to have a sample size more than 100 (Karasar, 
2012). We can state that 44% percent of the studies have an ‘advantageous’ or ‘ideal’ 


condition in terms of their sample size. 


Table 6: The Frequencies of the Sample Size 


Sample size f % 
0-10 14 19 
11-30 13 17 
31-60 8 11 
61-100 7 9 
101-500 23 31 
501-< 10 13 
TOTAL 75 100 


Classifying the studies in terms of their research design, the analysis revealed 
that 46% of the studies have qualitative research design, 41% of them have quantitative 
methods, and 13% of the studies have mixed method design. 


Table 7: The Frequencies of Research Design 


Research Design f % 
Qualitative 40 46 
Quantitative 35 41 
Mixed 11 113} 
TOTAL 86 100 


Compared to the other designs, there are fewer studies with a mixed design although 
mixed method (1) has the strong features of qualitative and quantitative designs, (2) can 


answer research questions with a larger scale, (3) can provide the insight and 
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understanding that a single design do not have, (4) and provides with the opportunity 
for the quantitative data to be interpreted visually and verbally and to digitize the 
qualitative data Johnson & Onwuegbuzie, 2004). The reasons behind the researchers’ 
choosing other designs over mixed method design can be due to the facts that (1) the 
workload can be too much for a researcher to handle alone, (2) that there is a need for 
the expertise to combine qualitative and quantitative designs, (3) and that researcher 
may need more time to conduct a mixed design research compared to other designs 
(JIohnson & Onwuegbuzie, 2004). 

Table 8 summarizes the ratio of the data collection tools. The analysis shows that 
there is a dominance of qualitative data collection tools such as interviews (32%), 
documents (13%), journals 81%), and field notes (2%). As nearly half of the studies 
(46%) have qualitative research designs, which is not surprising. Among the 
quantitative data collection tools, Likert-type scales are the ones that are mostly 
preferred by the researchers (25%). 25% of the studies used semi-structured interviews, 
5% of them used structured interviews, 1% used unstructured and focus group 
interviews. The reason behind the popularity of the semi structured interviews might 
be the belief that semi structured interviews provide a deeper insight and control over 
the responses (Karasar, 2012). 13% of the studies made use of documents to collect data. 
Among these documents were lesson plans of the teachers, reflections, textbooks, 
teachers’ guide, tests, and curriculum reports. While the observations (12%) were 
mainly used to check whether the classroom application of the curriculum was parallel 
with the way it was reported, achievement tests (5%) were used to determine the 


proficiency level of the students. 


Table 8: The Frequency of Data Collection Tools 


Data collection Tools Sub-category f % 

Open-ended [2 8 

Questionnaire Likert 4l 25 

Yes/No 2 1 

: Open-ended 2 1 
Achievement test RTSISGHOS 6 

Structured 8 5 

. Semi-structured 40 DS) 

Interview 

Unstructured 2 if 

Focus group D; 1 

Observation 19 12 

Documents | 13 

Others Journals 2 1 

Reports 1 1 

Field notes 4 2 


TOTAL 162 100 
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The issue to be highlighted here is that there is no experimental design among the ones 
quantitative studies. This shows us that the general tendency of the studies is to 
evaluate ELT curriculum through similar Likert-type scales from the perspectives of 
teachers and students. Investigating these scales that are mainly dealing with the 
opinions of English teachers and students of different grades about ELT curriculum, it 
is revealed that the main factors of the scales are goals, content, teaching/learning 
process and assessment, especially for the ones conducted in Turkey (Adigtizel, 2014; 
Alkan & Arslan, 2015; Amorim, 2010 Cankaya, 2015; Celen, 2011; Demirlier, 2010; Er, 
2006; Erkan, 2009; Inam, 2009; Kandemir, 2016; Lu, 1995; Ormeci, 2009; Oziidogru & 
Adigiizel, 2015; Sak, 2008; Topkaya & Kiiciik, 2010; Yanik, 2007; Yoru, 2012). 

Table 9 presents the data analysis techniques applied in the studies analyzed. 
The most frequently used analysis type is the content analysis (26%) followed by 
frequency and percentage (25%). The studies utilized t-tests (8%), ANOVAs (6%) as the 
parametric tests and Kruskal Wallis (2%), Mann Whitney U ( 2%) tests as the 
nonparametric ones to see whether there was a relation between independent variables 
such as gender, years of experience, taking an in-service training, the time spent in an 
English speaking country, age, the program graduated, and the opinions of the teachers 
on the curriculum (Abu-Ghararah, 1986; Altaieb, 2013; Burgos, 2012; Cankaya, 2015; 
Celen, 2011; Demirlier, 2010; Er, 2006; Erkan, 2009; Inam, 2009; Kershaw, 2009; Merter, 
Kartal & Caglar, 2012; Ormeci, 2009; Tom-Lawyer, 2014; Topkaya & Kticiik, 2010; Wang, 
2006; YG6rii, 2012). 


Table 9: The Frequency of Data Analysis 


Data Analysis f % 
Frequency/percentage 45 2) 
Mean/standard deviation 24 13 
Graphs 4 D 
T test I> 8 
Correlations 2 1 
ANOVA 10 6 
Regression 1 1 
Factor analysis 3 2 
Chi Square 6 3 
Kruskal Wallis | 2 
Mann Whitney U 3 2 
Cronbach Alpha | 2 
Content analysis 46 26 
Descriptive Analysis 13 7 
TOTAL 180 100 


Content analysis dealing with the findings of the studies has shown that regardless of 


the publication date, context, and grade, there were common issues that the studies 
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came up with. These shared findings were coded and the categories are presented in 
Figure 4. 


TEACHER (rans) 


-proficiency level -no information about 
cumiculum 

-lack of support to the student 
and teacher 


PRACTICAL ISSUES ASSESSMENT 


-class size -national exams 

-weekly course hours -skills /assessment tools 
-lack of m-service trammg -lack of examples 

-lack of need analysis -attamments 


-role in cummiculum design 


-no guidance 


-focus on grammar -inefficient / too much detail 
-too high ideals -overloaded 

-defining the goals -lack of interest 

-vagueness of directives -above the level of students 


Figure 4: Classification of the Common Findings 


As mentioned above the studies share a common pattern in terms of their findings no 
matter which grade they are dealing with or in which context they are conducted. These 
common findings can be classified under the themes of teacher, parents, assessment, 
practical issues, content, and material related issues. Starting with the practical issues, 
most of the studies were consistently stating that the weekly class hours of English 
lesson were not enough to achieve the goals of the curriculum. The relevant examples 
were the study by Donmez (2010), analyzing the 8" grade English curriculum, the one 
carried out by Erkan (2009) dealing with the curriculum of 4" grades, Karci’s study 
(2012) investigating the 9 grade curriculum in Turkey, the study of Al-Darwish (3006) 
evaluating the elementary school English curriculum of Kuwait, or the one conducted 
by Nakaprasit (2010) investigating ESL curriculum of a university in Canada. Moreover, 
the limited time given for English lessons caused teachers to neglect the four skills, 
especially speaking and listening, to focus more on grammar, and not to have enough 
time for in-class assessment (Giines, 2009; Kandemir, 2016; Kefeli, 2008; Yanik, 2007). 
The other findings under the practical issues were the crowded classrooms, lack of in- 
service training for the teachers, and lack of needs analysis (Altaieb, 2013; Cankaya, 
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2015; Donmez, 2019; Dudzik, 2008;Ege, 2006; Er, 2006; Erkan, 2009; Harris, 2010; 
lyitoslu & Alci, 2015; Jan, 1985; Karci, 2012; Kefeli, 2008; Mersinligil, 2002; Nam, 2005; 
Ormeci, 2009; Powell, 2008; Restivo, 2012; Seckin, 2010; Wang, 2006). 

The most commonly mentioned findings about the parents in the studies are that 
the parents had no idea about English curriculum; they were not supporting their 
children or did not have a good communication with the teachers (Burgos, 2013; 
Krekeler, 1993; Restivo, 2012; Wu, 2013; Yanik, 2007). When it comes to the teacher- 
related findings, it was revealed that the teachers were not proficient enough especially 
to use English as a medium of instruction during the class hours, they had no 
responsibility during the curriculum design, and their opinions were not valued by the 
program facilitators (Almalki, 2014; Alwan, 2006; Kim, 2008; Lundien, 2009; Nunan, 
2003; Tsai, 2007; Zaid, 1993). 

Other common points mentioned in the studies were the negative impact of the 
public examinations on the application of the English curriculum and the lack of 
assessment tools and examples, especially for measuring the listening and speaking 
skills. It was found out that public examinations forced English teachers to base their 
teaching on training students to get high marks from the public tests which put 
pressure on the teachers (Alkan & Arslan, 2015; Donmez, 2010; Inam, 2009; Gunes, 2009; 
Jan, 1985; Karakoyun, 2008; Kim, 2008; Nonthaisong, 2015; Tsai, 2007; Wang, 2006; Wu, 
2013; Yigit, 2010). 

Some common findings on the content of the curriculum were also revealed as a 
result of the analysis. It was stated that the directives of the curricula were not clear, 
therefore, every teacher conducted their lessons based on the perspective they got form 
those vague directions. Moreover, it was mentioned that the focus of the curriculum 
was on grammar and the attainments were too ideal for a classroom environment and 
above the level of the students (Al-Darwish, 2006; Carroll, 2005; Erdogan, 2005; Erkan; 
2009; Glasgow, 2014; Hillberry, 2008; Hu, 2007; Kandemir, 2016; Kershaw, 2009; Lai, 
2007; Nunan, 2003; Oraka, 2012; Ormeci, 2009; Restivo, 2012; Sak, 2008; Wang, 1996; 
Wang, 2006; Yanik, 2007). 

Finally, it was mentioned by most of the studies that the textbooks and teachers’ 
guides were inefficient and were irrelevant to the curriculum itself. There was a need 
for extra material and equipment, or in some situations, those textbooks could be seen 
as the curriculum itself. It was also claimed that the content of the materials were 
overloaded and not prepared in line with the interest and level of the students 
(Cankaya, 2015; Demir & Duruhan, 2015; Dénmez, 2010; Ege, 2006; Er, 2006; Giines, 
2009; Jan, 1985; Karakoyun, 2008; Kticiiktepe, Kiictiktepe, & Baykin, 2014; Mersinligil, 
2002; Seckin, 2010; Tom-Lawyer, 2014; Yaman, 2010; Yanik, 2007; Yildiran & Tanrisever, 
2015; Y6rti, 2012; Zaid, 1993). 
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The underlying reason for the common findings obtained by the aforementioned 
studies might result from the fact that the policy of ESL/EFL contexts have changed 
through the time, mainly at the approach level. It is evident that findings are mainly on 
the practical issues as the policy and curriculum mainly do not deal with the practical 
sides such as the classroom size, the supportive materials, or the proficiency levels of 
the teachers. In other words, as the implementation challenges have remained and the 
curricula designed in ESL/EFL contexts do not pay attention to those challenges, it is not 
surprising that teachers, especially the ones in Turkey, are stating the same opinions 


again and again. 
4. Conclusions 


The present study intends to specify the tendency of the research on English language 
curriculum development and evaluation in ESL/EFL contexts. Main conclusions drawn 
from the results obtained in this analysis are as follows: 

1. The studies have mainly focused on curriculum evaluation rather than 
curriculum development. 

2. There is a relation between the changes in the education policy of the countries 
and the grades evaluated. 

3. The studies generally concerned with the opinions of English teachers and 
students. 

4. Likert-type scales, semi-structured interviews and documents have been the 
main data collection tools. 

5. The studies presented common findings regardless of their context, publication 
time and the grade evaluated. 

This study can provide a base for further studies and for a better investigation of 
the research on curriculum development and evaluation in ESL/EFL as it shows the 
general profile of the studies and reveals the points that have not been covered yet. 
Thus, the study can show the researchers what has been done so far and the points to be 
researched. 

Being the first of its kind, this study is able to provide future researchers with the 
following suggestions: 

1. In any context, curriculum development studies should be increased. 

2. The studies should not only deal with the opinions of the teachers or students, 
but also the opinions or experiences of the parents, program facilitators, 
academic staff, and officials. Teachers are not the only stakeholders of the 
curriculum design and evaluation process. In order to have detailed information 


on the design and implementation of English language curriculum, more studies 
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are needed considering the opinions of the other stakeholders like students’ 

parents, and educational administrators. 

3. Researchers should pay attention to reliability and validity of the data collection 
instruments. 

4. Researchers can also attend to the issue of in-service training on the 
implementation of the curriculum. 

5. Experimental studies can also be conducted in order to develop an English 
language curriculum and measure its efficacy. 

6. Meta-analysis studies can be conducted in order to calculate the effect size of 
gender, experience, in-service training, and grades on the evaluation of the 
curriculum. 

7. The number of critical analysis studies should be increased in the field of English 
language teaching. 

As a last word, the present study is limited to 86 studies conducted in ESL/EFL 
contexts from 1985 to 2016 on the issue of curriculum development and evaluation. 
Only the articles and theses available online were included in this review. The studies 
that are not available online, conference papers, and or book chapters were not 
included. Therefore, the results of the study should be considered and generalized 


bearing these limitations in mind. 
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APPENDIX 

List of the studies taken for the critical review 

Author(s) Date | Publication Type Country Grade/Level Focus 
Jan, M. A. 1984 PhD Diss. Saudi Arabia Intermediate boy’s school CE 
Abu-Ghararah, A. H. 1986 PhD Diss. Saudi Arabia Secondary School CE 
Kaewsanchai, N. 1988 PhD Diss. Thailand College CE 
Krekeler, C. F. 1993 PhD Diss. USA Elementary level CE 
Lu, J. 1995 PhD Diss. China College CE 
Sowers, J. L. 1996 PhD Diss. Japan Kindergarten CD 
Wang, L. H.C. 1996 PhD Diss. Taiwan High School CE 
Karatas, N., & Tiirkoglu, A. 1997 Article Turkey College CE 
Fang, Y. 2002 PhD Diss. Taiwan Primary and secondary level CE 
Mersinligil, G. 2002 PhD Diss. Turkey 4™ and 5" Grades CE 
Nunan, D. 2003 Article Asia Pacific Region All levels CE 
Sart, R. 2003 PhD Diss. Turkey College CD 
Haznedar, B. 2004 Article Turkey Primary level CE 
Biiyiikduman, F. I. 2005 Article Turkey Primary CE 
Carroll, K. S. 2005 MA Thesis Puerto Rico Secondary School CE 
Erdogan, V. 2005 MA Thesis Turkey 4" and 5" Grades CE 
Gerede, D. 2005 MA Thesis Turkey College CE 
Nam, J. M. 2005 PhD Diss. Korea College CE 
Zaid, M. A. 2005 PhD Diss. Saudi Arabia Intermediate School CE 
Al-Darwish, S. 2006 PhD Diss. Kuwait Elementary School CE 
Alwan, F. H. 2006 PhD Diss. United Arab Emirates Secondary School CE 
Ege, I. 2006 MA Thesis Turkey College CE 
Er, K. 2006 PhD Diss. Turkey 4" and 5™ Grade CE 
Wang, H. 2006 PhD Diss. China College CE 
Hu, Y. 2007 PhD Diss. China Primary School CE 
Lai, C. C. 2007 PhD Diss. Taiwan Elementary school CE 
Sun, L. 2007 PhD Diss. Taiwan From 1“ to 9™ Grades CE 
Tsai, T. H. 2007 PhD Diss. Taiwan Junior high school CE 
Yanik, A. 2007 PhD Diss. Turkey 6", 7" and 8" Grades CE 
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Dudzik, D. 2008 PhD Diss. Djibouti Middle School CE 
Hillberry, M. M. 2008 PhD Diss. USA Elementary level CE 
Karakoyun, S. 2008 MA Thesis Turkey Secondary level CE 
Kefeli, H. 2008 PhD Diss. Turkey High school CE 
Powell, J. M. 2008 PhD Diss. USA Middle school CE 
Sak, O. 2008 MA Thesis Turkey Primary level CE 
Spencer, C. H. T. S. 2008 PhD Diss. Taiwan College CE 
Erdem, A. 2009 Article Turkey, Ireland Primary and secondary level CE 
Erkan, M. A. 2009 MA Thesis Turkey 4" and 5" Grades CE 
Giines, T. 2009 MA Thesis Turkey 5" Grade CE 
Inam, G. 2009 MA Thesis Turkey 4" Grade CE 
Kerdshaw, P. J. 2009 PhD Diss. USA Middle school CE 
Lundien, K. 2009 PhD Diss. USA Secondary level CE 
Nakaprasit, T. 2009 MA Thesis Canada College CE 
Ormeci, D. 2009 MA Thesis Turkey 4" 5™ and 6" Grades CE 
Perez, A. N. 2009 MA Thesis Puerto Rico Kindergarten CD 
Amorim, G. B. 2010 PhD Diss. Brazil College CE 
Demirlier, H. 2010 MA Thesis Turkey Primary school CE 
Dénmez, O. 2010 PhD Diss. Turkey 8"" Grade CE 
Harris, L. S. 2010 PhD Diss. USA High School CE 
Seckin, H. 2010 PhD Diss. Turkey 4" Grade CE 
Topkaya, E. Z., & Kiiciik, O. 2010 Article Turkey 4" and 5" Grades CE 
Yaman, S. 2010 MA Thesis Turkey Primary level CE 
Yidit, C. 2010 MA Thesis Turkey 6" Grade CE 
Celen, G. 2011 MA Thesis Turkey 6" Grade CE 
Qiao, X. 2011 PhD Diss. China College CE 
Tucker, T. 2011 Article Korea College CE 
Burgos, S. 2012 PhD Diss. Puerto Rico Elementary ,Junior, High School CE 
Kare, C. 2012 MA Thesis Turkey 9" Grade CE 
Merter, F., Kartal, $., & Caélar, I. 2012 Article Turkey Secondary level CE 
Orakci, S. 2012 MA Thesis Turkey 7" Grade CE 
Ozer, O. 2012 MA Thesis Turkey 8"" Grade CE 
Restivo, S. 2012 PhD Diss. USA High School CE 
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Yorii, B. 2012 MA Thesis Turkey 8" Grade CE 
Altaieb, S. R. 2013 PhD Diss. Libya High School CE 
Wu, W. 2013 PhD Diss. China Junior high school CE 
Adigiizel, O. C., & Oziidogru, F. 2014 Article Turkey 2™ Grade CE 
Almalki, M. M. 2014 PhD Diss. Saudi Arabia Secondary School CE 
An, A. 2014 Article Turkey 6" Grade CE 
Demir, Y. & Yavuz, M. 2014 Article Turkey, Finland, Japan, | Primary and secondary level CE 
Korea, China 
Glasgow, G. P. 2014 Article Japan Senior high school CE 
Kim, E. A. 2014 PhD Diss. Korea Kindergarten CE 
KozikoSlu, I. 2014 Article Turkey 7" Grade CE 
Kictiktepe, C., Kiictiktepe, S. E., & Baykin, Y. 2014 Article Turkey 2" Grade CE 
Mavis, F. O., & Bedir, G. 2014 Article Turkey 2" Grade CE 
Tom-Lawyer, O. 2014 Article Nigeria College CE 
Alkan, M. F., & Arslan, M. 2015 Article Turkey 2" Grade CE 
Aybek, B. 2015 Article Turkey 2"° Grade CE 
Cankaya, P. 2015 MA Thesis Turkey 3" Grade CE 
Demir, O., & Duruhan, K. 2015 Article Turkey 2" Grade CE 
Demirtas, Z., & Erdem, S. 2015 Article Turkey 5" Grade CE 
lyitoglu, O., & Alci, B. 2015 Article Turkey 2" Grade CE 
Nonthaisong, K. 2015 PhD Diss. Thailand Secondary level CE 
Oziidogru, F., & Adigiizel, O. C. 2015 Article Turkey 2" Grade CE 
Yildiran, C., & Tanriseven, I. 2015 Article Turkey 2” Grade CE 
Kandemir, A. 2016 MA Thesis Turkey 2" grade CE 
Zorba, M. G., & Arikan, A. 2016 Article Turkey 9" Grade CE 
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